Add RFC: Road to Native Wayland (Linux) #14

GeorgesStavracas · 2020-03-09T15:10:12Z

Start Date: 2020-03-09
RFC PR: Add RFC: Road to Native Wayland (Linux) #14
Mantis Issue: N/A

Summary

Add native Wayland support for OBS Studio

Motivation

Wayland is a big and widespread alternative to the X11 protocol. Most desktop environments on Linux support Wayland now, and it is the default on GNOME. There are also many Wayland-only window managers being created every day.

OBS Studio has X11-specific code, which makes it incompatible with Wayland.

Detailed design

This RFC covers both making OBS Studio be able to run as a native Wayland application, and also adding screen and window captures that work on Wayland compositors.

Native Client (✓ done)

I hereby propose a 3-step approach to achieve native Wayland integration:

Introduce an EGL-X11 renderer to libobs-opengl (✓ done)
Add the concept of a Wayland platform to libobs (✓ done)
Introduce an EGL-Wayland renderer to libobs-opengl (✓ done)

Optionally, a 4th step may be:

Add an experimental checkbox to the Configuration dialog to enable EGL;

A detailed explanation of each step follows below.

EGL-X11 (✓ done)

This step involves adding an abstraction layer to the windowing system, renaming libobs-opengl/gl-x11.c to libobs-opengl/gl-glx.c, and introducing a new libobs-opengl/gl-egl-x11.c file.

The glad dependency would also be updated to have the necessary EGL files.

The abstraction layer is composed of libobs-opengl/gl-nix.{c|h}. This is where the exported functions (i.e. everything included in struct gs_exports) are implemented. All these function implementations, however, will look like this:

extern void gl_foo_bar(...)
{
	gl_winsys->foo_bar(...);
}

That's because gl_winsys is selected runtime, and may be either the GLX winsys, or the EGL/X11 winsys.

Wayland Platform (✓ done)

The Wayland Platform step introduces a way to detect at runtime which platform OBS Studio is running on. Initially, only Unix implementations will set and actually use it.

The header file would look like:

libobs/obs-nix-platform.h

enum obs_nix_platform_type {
	OBS_NIX_PLATFORM_X11_GLX,
	OBS_NIX_PLATFORM_X11_GLX,
	OBS_NIX_PLATFORM_WAYLAND,
};

EXPORT void obs_set_nix_platform(enum obs_nix_platform_type platform);
EXPORT enum obs_nix_platform_type obs_get_nix_platformobs_get_nix_platform(void);

EXPORT void obs_set_nix_platform_display(void *display);
EXPORT void *obs_get_nix_platform_display(void);

EGL-Wayland (✓ done)

The 3rd and last step is introducing the libobs-opengl/gl-egl-wayland.{c|h} files, and making gl-nix retrieve the EGL/Wayland winsys. This is the only winsys that can be used when running under Wayland.

Screen & Window Capture

The process for capturing screen and window contents on Wayland compositors is different than on Xorg. While X11 gives applications the ability to spy on anything, including other applications, the entire compositor, Wayland proposes a much stricter security model.

Portals

On Wayland, the general way of requesting the compositor for sharing the screen or window contents is through the ScreenCast portal.

Portals are D-Bus interfaces that applications use to interact with the desktop. The ScreenCast portal specifically provides applications the necessary tools to ask the desktop environment to share the contents of a window or a monitor.

Known to implement the ScreenCast portal is GNOME, KDE, and wlroots. These three should cover the majority of Wayland compositors in existence today.

PipeWire

The ScreenCast portal is written based on PipeWire to work. PipeWire is a service that provides a robust and performant media sharing mechanism for video and audio.

There already is an out-of-tree plugin, obs-xdg-portal, that implements this. It can serve as a basis for either a new in-tree plugin, or be incorporated as part of the linux-capture plugin.

How We Teach This

N/A, this simply changes the current behaviour. As long as the patch notes properly describe the changes, we should be fine.

Drawbacks

This doesn't change the default behavior of OBS Studio, which reduces the surface area for regressions.

Alternatives

Not supporting Wayland?

Unresolved questions

Should the capture code be a new in-tree plugin, or incorporated into linux-capture?

GeorgesStavracas · 2020-03-09T15:18:48Z

From Discord:

can vulkan be used? just sort of curious about that

Not sure, but I believe it can be. Writing Vulkan clients that work on Wayland is relatively easy and well tested, but I'm not sure what happens when QT joins the party.

It's not in the scope of this RFC (Vulkan doesn't depend in any way on Wayland), so it's probably the case for another RFC specifically for that.

Additionally, I don't know how mature is Vulkan support for DMA-BUF importing, but I assume it is pretty new and untested.

jp9000 · 2020-04-20T22:39:18Z

I think implementing Vulkan specifically for use with Linux would likely provide useful features to us in the future, but should probably be a separate discussion.

I don't generally have a problem with implementing wayland support if there are no downsides and it doesn't interfere with non-wayland cases. The only problem with wayland is mostly "how do you capture a display/desktop", or "how do you capture a window".

jp9000 · 2020-04-20T23:03:02Z

So let me actually rephrase this as questions:

Does this have any downside, or does it affect non-wayland use cases in any way?
How does capture work with wayland? Both display capture and window capture.

GeorgesStavracas · 2020-04-20T23:17:35Z

Those are valid concerns, thanks for bring them up. I'm happy to clarify them 🙂

Does this have any downside, or does it affect non-wayland use cases in any way?

No, it absolutely doesn't affect non-Wayland.

This RFC, and the PR implementing it, is opt-in: people will have to explicitly request OBS Studio to run as a native Wayland client, otherwise it always uses the X11 code paths.

How does capture work with wayland? Both display capture and window capture.

Sort answer: the ScreenCast interface of the Desktop portal.

Long answer:

There is an overall trend that was born inside the sandboxing technologies (Flatpak & Snap) of moving towards what are called "portals". Portals are well defined D-Bus APIs that applications should use in order to integrate and request stuff from the host system. Portals match the philosophy of Wayland, so it's became the de facto standard now.

One of these portals is the Desktop portal, which has various utilities for the desktop use case. For our case, the relevant one is the ScreenCast interface, which is precisely about screencasting.

Currently, the ScreenCast interface is implemented by GNOME, KDE, and wlroots, which covers the vast majority of Wayland compositors.
(Keep in mind, though, that the portals are not related to Wayland: they work just as fine on X11, or Xwayland. They also work just fine when running inside and outside a sandbox.)

The ScreenCast interface is built around PipeWire. I wrote a very rough OBS plugin around this portal, and PipeWire, the obs-xdg-portal plugin, and it's working really well.

Let me know if you have more questions or concerns!

GeorgesStavracas · 2020-04-20T23:53:11Z

Forgot to mention, but the ScreenCast interface covers both the monitor and window capture cases. Using the aforementioned plugin, the Desktop portal asks what the user wants to capture:

The compositor (GNOME, KDE, wlroots) then shares the appropriate contents with OBS Studio.

kkartaltepe · 2020-04-21T02:06:54Z

For the moment xdg-desktop-portal (screencasting extension, not the other 12 apis in this standard) is probably the only way to write one piece of code that will mostly work on most platforms. But it sure wont be fast, and users will undoubtedly still rely on third party per-DE plugins that implemented the per-DE wayland protocols for zero copy textures (and maybe one day nvidia will join the fun). But until we provide an example of a good program for users to show how awful the platform is, the DEs have no motivation to standardize on anything good for users (arguably we are probably too late in this respect. At least I too much faith in DEs left to their devices). In this sense its important to support even if our capture support is lacking.

There are already plugins for every major DE already in wide circulation, so lets let them finally build against an upstream target instead of forks.

Portals match the philosophy of Wayland, so it's became the de facto standard now.

What a joke... This is gnome literally giving up on wayland protocols in order to use dbus and find a use for pipewire (arguably a good idea for gnome). It's a complete slap in the face of wayland.

GeorgesStavracas · 2020-04-21T02:40:56Z

But it sure wont be fast, and users will undoubtedly still rely on third party per-DE plugins that implemented the per-DE wayland protocols for zero copy textures

Why would it not be fast? It is possible to implement zero-copy buffer sharing on top of PipeWire. GNOME implements that, and as far as I'm aware, wlroots folks are working towards that as well.

What a joke... This is gnome literally giving up on wayland protocols in order to use dbus and find a use for pipewire (arguably a good idea for gnome). It's a complete slap in the face of wayland.

Those are... harsh words, and I don't think that's correct either. GNOME is not giving up on Wayland protocols.

Screen casting is an example of a feature that requires more than a low-level protocol standard in order to make sense. Most Wayland protocols are built around the idea that applications hand compositors their buffers; screencasting is the exact opposite of that. Screen casting also has privacy implications, and needs a security model around it. Compositors can't just let every application to be able to have access to their buffers. Applications can freely access the Wayland socket, and screen capturing on the protocol level requires at least an authentication mechanism, which is outside the scope. At last, doing that on Wayland level forces compositors to step in "multimedia" territory, which is also not something compositors usually aim to do.

Finally, I'm not sure why you think GNOME is trying to find a use for PipeWire. PipeWire is built for this use-case, and is not a GNOME-specific project. KDE and wlroots use it for screen casting as well.

dodgepong · 2020-04-21T04:09:06Z

I think it would behoove us to stay focused on what we can actually do, practically speaking, to have Wayland support. It may not be ideal, and it may even be frustrating or unfortunate at points, but it's better to have something rather than nothing.

mb720 · 2020-04-21T08:47:20Z

Thanks a lot for working on Wayland support!

I just wanted to make sure you've heard about wf-recorder: I use it currently on Sway to record the screen and audio. It works flawlessly.

I don't believe wf-recorder uses the Desktop portal, though.

Unveiled53 · 2020-05-11T03:24:25Z

So I've been testing OBS with these PRs for a couple hours, and it seems to be working pretty good. Only noticeable bug I've encountered so far is on the property windows, it makes a transparent "hole" in the window (Like the hole the preview window would get before these PRs), and it gets fixed only when I resize it with the 6 dots. My setup is 5700 XT + sway (wlroots) + 2 4k monitors scaled 1.5x + 1080p monitor. Otherwise, the PRs work great and I'm glad this work is being done

mvdan · 2020-10-25T21:28:50Z

Thank you so much for this large amount of work, @GeorgesStavracas. Do you accept donations?

lanodan · 2020-12-28T05:14:42Z

Thanks for the work done so far, it's great to be able to use OBS in wayland.

As for the buildsystem, I think it should be something like WINDOWING_SYSTEMS=x11,wayland instead of ENABLE_WAYLAND, this way the wayland part of obs could be used without X11 libs being present, at least in the future. Otherwise maybe have an ENABLE_X11 equivalent and error out if none of wayland and X11 are enabled on non-apple unix.

Fenrirthviti · 2021-03-30T16:46:54Z

Most of this work has been completed and merged in to OBS, are there any remaining updates that need to be made before this can be merged?

Entering fast-track FCP with disposition merge. Will merge at EOD unless there are any strong objections.

GeorgesStavracas · 2021-03-30T18:42:12Z

I believe all that needed to be done here already landed. From now on, it's mostly about polishing this code, bugfixing, and improving the stack to better support OBS Studio's use case - I'm not sure if that's worth a new RFC.

dodgepong mentioned this pull request Apr 26, 2020

[linux] OBS Studio running on Wayland session of KDE Plasma desktop environment has Wayland generic icon in its title bar and in alt+tab menu obsproject/obs-studio#2585

Closed

Fenrirthviti mentioned this pull request Jun 26, 2020

Cannot capture entire screen in Wayland obsproject/obs-studio#2471

Closed

GeorgesStavracas force-pushed the master branch 2 times, most recently from 81439db to 50a292f Compare February 13, 2021 23:24

Add RFC: Road to Native Wayland (Linux)

5244537

GeorgesStavracas force-pushed the master branch from 50a292f to 5244537 Compare February 13, 2021 23:31

Fenrirthviti added disposition merge final comment period labels Mar 30, 2021

Fenrirthviti merged commit 1eef8a5 into obsproject:master Mar 31, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add RFC: Road to Native Wayland (Linux) #14

Add RFC: Road to Native Wayland (Linux) #14

GeorgesStavracas commented Mar 9, 2020 •

edited

Loading

GeorgesStavracas commented Mar 9, 2020

jp9000 commented Apr 20, 2020

jp9000 commented Apr 20, 2020

GeorgesStavracas commented Apr 20, 2020

GeorgesStavracas commented Apr 20, 2020

kkartaltepe commented Apr 21, 2020 •

edited

Loading

GeorgesStavracas commented Apr 21, 2020

dodgepong commented Apr 21, 2020

mb720 commented Apr 21, 2020

Unveiled53 commented May 11, 2020 •

edited

Loading

mvdan commented Oct 25, 2020

lanodan commented Dec 28, 2020 •

edited

Loading

Fenrirthviti commented Mar 30, 2021

GeorgesStavracas commented Mar 30, 2021

Add RFC: Road to Native Wayland (Linux) #14

Add RFC: Road to Native Wayland (Linux) #14

Conversation

GeorgesStavracas commented Mar 9, 2020 • edited Loading

Summary

Motivation

Detailed design

Native Client (✓ done)

EGL-X11 (✓ done)

Wayland Platform (✓ done)

EGL-Wayland (✓ done)

Screen & Window Capture

Portals

PipeWire

How We Teach This

Drawbacks

Alternatives

Unresolved questions

GeorgesStavracas commented Mar 9, 2020

jp9000 commented Apr 20, 2020

jp9000 commented Apr 20, 2020

GeorgesStavracas commented Apr 20, 2020

GeorgesStavracas commented Apr 20, 2020

kkartaltepe commented Apr 21, 2020 • edited Loading

GeorgesStavracas commented Apr 21, 2020

dodgepong commented Apr 21, 2020

mb720 commented Apr 21, 2020

Unveiled53 commented May 11, 2020 • edited Loading

mvdan commented Oct 25, 2020

lanodan commented Dec 28, 2020 • edited Loading

Fenrirthviti commented Mar 30, 2021

GeorgesStavracas commented Mar 30, 2021

GeorgesStavracas commented Mar 9, 2020 •

edited

Loading

kkartaltepe commented Apr 21, 2020 •

edited

Loading

Unveiled53 commented May 11, 2020 •

edited

Loading

lanodan commented Dec 28, 2020 •

edited

Loading